Towards accessible content creation of real world objects for virtual environments

3D reconstruction is the general problem of creating 3D models from real world objects. In today\u27s movie and games industry,there is an increasing demand for using real world content as assets in production. In general, however, 3D reconstruction is achallenging problem, and current techniques only allow for production-ready results given a combination of expensive equipment andspecific expertise.This thesis is a collection of three papers that address various aspects of this general problem of 3D reconstruction,with the aim of lowering the bar for making usable real world content.In Paper I, we address the problem of storing and streaming time varying geometry for e.g.\ free-viewpoint video, whichotherwise has too high bandwidth requirements to be streamed efficiently. We use a memory-efficient structure based on compressedvoxels to store the data, in which we can send only incremental updates to the geometry in each frame.In Paper II, we implement an end-to-end real-time pipeline for free-viewpoint video communication.The pipeline uses a set of ordinary webcams as input and do all processing on a single desktop computer. Even with theselimitations, we show that we can produce free-viewpoint video with agreeable quality in real-time.Paper III addresses the problem of accessible and accurate modeling of static real-world objects.Given a set of calibrated input images, we have developed an interactive tool that makes 3D reconstruction with multi-view stereo moreaccessible. This interactive reconstruction has several advantages over automatic 3D scanning, since we obtain correct topology by designas well as information about visibility and foreground segmentation

Automatic Test Methods for Image and Video Verification

In this thesis four methods for automatic verification of images and video on mobile platforms are developed. Both the case of recording images and video and the case of viewing images and video on the mobile lcd screen are considered. The first method is used to test the zoom function of the camera. It uses SURF decriptors along with clustering and histograms to determine which of six discrete zoom levels the current frame belongs to. The second method identifies color effects and color anomalies using histograms. The third method determines if the autofocus works correctly by measuring the average length of edges in the image. The fourth method is an artifact detection scheme using a non-reference implementation of the SSIM metric, used in conjunction with a for this purpose specially designed test setup. Together these methods form a tool kit for detecting the mnost common errors to occur in images and video during the development stage of mobile platforms

PERF: Performant, Explicit Radiance Fields

We present a novel way of approaching image-based 3D reconstruction based on radiance fields. The problem of volumetric reconstruction is formulated as a non-linear least-squares problem and solved explicitly without the use of neural networks. This enables the use of solvers with a higher rate of convergence than what is typically used for neural networks, and fewer iterations are required until convergence. The volume is represented using a grid of voxels, with the scene surrounded by a hierarchy of environment maps. This makes it possible to get clean reconstructions of 360\ub0 scenes where the foreground and background is separated. A number of synthetic and real scenes from well-known benchmark-suites are successfully reconstructed with quality on par with state-of-the-art methods, but at significantly reduced reconstruction times

A low-cost, practical acquisition and rendering pipeline for real-time free-viewpoint video communication

We present a semiautomatic real-time pipeline for capturing and rendering free-viewpoint video using passive stereo matching. The pipeline is simple and achieves agreeable quality in real time on a system of commodity web cameras and a single desktop computer. We suggest an automatic algorithm to compute a constrained search space for an efficient and robust hierarchical stereo reconstruction algorithm. Due to our fast reconstruction times, we can eliminate the need for an expensive global surface reconstruction with a combination of high coverage and aggressive filtering. Finally, we employ a novel color weighting scheme that generates credible new viewpoints without noticeable seams, while keeping the computational complexity low. The simplicity and low cost of the system make it an accessible and more practical alternative for many applications compared to previous methods

PERF: Performant, Explicit Radiance Fields

We present a novel way of approaching image-based 3D reconstruction based on radiance fields. The problem of volumetric reconstruction is formulated as a non-linear least-squares problem and solved explicitly without the use of neural networks. This enables the use of solvers with a higher rate of convergence than what is typically used for neural networks, and fewer iterations are required until convergence. The volume is represented using a grid of voxels, with the scene surrounded by a hierarchy of environment maps. This makes it possible to get clean reconstructions of 360\ub0 scenes where the foreground and background is separated. A number of synthetic and real scenes from well-known benchmark-suites are successfully reconstructed with quality on par with state-of-the-art methods, but at significantly reduced reconstruction times

Exploiting coherence in time-varying voxel data

We encode time-varying voxel data for efficient storage and streaming. We store the equivalent of a separate sparse voxel octree for each frame, but utilize both spatial and temporal coherence to reduce the amount of memory needed. We represent the time-varying voxel data in a single directed acyclic graph with one root per time step. In this graph, we avoid storing identical regions by keeping one unique instance and pointing to that from several parents. We further reduce the memory consumption of the graph by minimizing the number of bits per pointer and encoding the result into a dense bitstream

Techniques for Fast and High-Quality 3D Reconstruction of General Scenes

This thesis is a collection of techniques used for 3D reconstruction; the creation of 3D models fromreal world objects or scenes. Given the increase in accuracy, robustness and speed of modern methodsand algorithms, new and exciting applications of this technology is constantly appearing. Asset creationfor games and movies is a successful example, but there are numerous other applications in architecture,medicine, communication and more.The contribution of each paper in this thesis aims to make the use of 3D reconstruction even more ubiquitous by addressing problems such as performance, memory usage, ease-of-use, robustnessand quality.Paper I presents a compression technique for volumetric video modeled with voxels. Memory consumption is an important issue when storing volume data, especially if the data is also varyingwith time. Paper II describes an end-to-end pipeline for recording and rendering volumetric video. Asimple and readily available setup of webcams and a single desktop computer is used to record and renderscenes in real-time.In Paper III, an interactive tool is developed that aims to help in modeling of real-world objects. Structured as a simple quad modeling program, the user can construct 3D models on top of aset of photographs of a chosen object. In the background, or after explicit activation, a multi-viewstereo algorithm helps the user to align the geometry correctly to images in world space. This greatlysimplifies the problem of modeling real world objects accurately, while levering the input from the userto help with topology and visibility.Paper IV implements a direct solver for the problem of neural rendering. The reconstructionis formulated as a non-linear least-squares problem which is solved efficiently with the Gauss Newtonmethod and the Preconditioned Conjugate Gradient algorithm. This formulation achieves a significantimprovement to reconstruction times compared to previous methods, while also being suitable for distributed computing due to needing three order of magnitudes fewer iterations until convergence.Paper V handles the shape-radiance ambiguity in neural rendering. Given infinite spatialresolution of view-dependent information, almost any shape can satisfy the incoming radiance to each camera, resultingin errors in the geometry. To address this problem, we propose a solution to separate Lambertian and view-dependent colorsduring reconstruction